Image editing device and image editing method

ABSTRACT

An image editing device of the present invention, for editing a combined photographic image formed by combining a plurality of images, comprises a first image processing section for carrying out first image processing on the image and creating a first image, a second image processing section for carrying out second image processing on the first image to create a second image, a combined photograph processing section for creating a combined photograph by combining the second image in accordance combining position of the combined photograph, and a control section for, after the first image processing has been carried out, creating a combined photograph by combining the second image that has been subjected to the second image processing in accordance with combining position of the combined photograph by the combined photograph processing section, and storing the combined photograph and the first image as a single image file.

Benefit is claimed, under 35 U.S.C. §119, to the filing date of priorJapanese Patent Application No. 2012-213258 filed on Sep. 26, 2012. Thisapplication is expressly incorporated herein by reference. The scope ofthe present invention is not limited to any requirements of the specificembodiments described in the application.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image editing device for editing animage such as a combined photograph made up of a plurality ofphotographs, and to an imaging editing method.

2. Description of the Related Art

Conventionally, there have been various approaches to carrying outgradation processing for digital image signals, in order to achieveimage storage without gradation jumps (refer, for example, to Japanesepatent laid-open No. 2005-051337 and Japanese patent laid-open No. Hei.09-233339). Also, since there is gradual deterioration if an image isrepeatedly used, deterioration prevention measures are implemented(refer, for example, to Japanese patent laid-open No. 2010-276765 andJapanese patent laid-open No. 2001-143055).

SUMMARY OF THE INVENTION

An object of the present invention is to provide an image editing deviceand image editing method with which there is mo image deterioration,even if combined photograph editing to replace recombined an image ofpart of a combined photograph is carried out repeatedly.

An image editing device of the present invention, for editing a combinedphotographic image formed by combining a plurality of images, comprisesa first image processing section for carrying out first image processingon the image and creating a first image, a second image processingsection for carrying out second image processing on the first image tocreate a second image, a combined photograph processing section forcreating a combined photograph by combining the second image inaccordance combining position of the combined photograph, a storagesection for storing the first image and an image of the combinedphotograph, and a control section for, after the first image processinghas been carried out, creating a combined photograph by combining thesecond image that has been subjected to the second image processing inaccordance with combining position of the combined photograph by thecombined photograph processing section, and storing the combinedphotograph and the first image as a single image file.

An image editing method of the present invention, for editing a combinedphotographic image formed by combining a plurality of images, comprisesa step of carrying out first image processing on the image and creatinga first image, a step of, after creating the first image, carrying outsecond image processing on the first image and creating a second image,a step of creating a combined photograph by combining the second imagein accordance with combining position of the combined photograph, and astep of storing the combined photograph and the first image as a singleimage file.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram mainly showing the electrical structure of acamera of one embodiment of the present invention.

FIG. 2 is a drawing showing one example of image-editing, in a camera ofone embodiment of present invention, and shows the appearance ofcarrying out image editing by touching a preview.

FIG. 3 is a drawing showing one example of image-editing, in a camera ofone embodiment of present invention, and shows the appearance ofcarrying out image editing by touching a preview.

FIG. 4 is a drawing showing one example of image-editing, in a camera ofone embodiment of present invention, and shows the appearance ofdeciding an editing frame.

FIG. 5 is a drawing showing an example of screen division, with thecamera of one embodiment of the present invention.

FIG. 6 is a flowchart showing main operation of the camera of oneembodiment of the present invention.

FIG. 7 is a flowchart showing main operation of the camera of oneembodiment of the present invention.

FIG. 8 is a flowchart showing operation of image processing of thecamera of one embodiment of the present invention.

FIG. 9 is a flowchart showing operation of basic image processing of thecamera of one embodiment of the present invention.

FIG. 10 is a flowchart showing operation of special image processing ofthe camera of one embodiment of the present invention.

FIG. 11 is a flowchart showing operation of combined photographformation of the camera of one embodiment of the present invention.

FIG. 12A-FIG. 12C are drawings showing a modified example of imageanalysis, with the camera of one embodiment of the present invention.

FIG. 13 is a flowchart showing operation of still image storage for thecamera of one embodiment of the present invention.

FIG. 14 is a flowchart showing operation of combined photographmanipulation of the camera of one embodiment of the present invention.

FIG. 15 is a flowchart showing operation of combined photographmanipulation of the camera of one embodiment of the present invention.

FIG. 16A-FIG. 16C are drawings of the structure of combined photographmanipulation memory, with the camera of one embodiment of the presentinvention.

FIG. 17 is a flowchart showing playback and editing operations of thecamera of one embodiment of the present invention.

FIG. 18 is a flowchart showing operation of combined photograph editingof the camera of one embodiment of the present invention.

FIG. 19 is a flowchart showing operation of combined photographformation and storage of the camera of one embodiment of the presentinvention.

FIG. 20 is a drawing showing a file structure before combining of thecombined photograph, in a camera of one embodiment of the presentinvention.

FIG. 21 is a drawing showing a file structure before combining of thecombined photograph, in a camera of one embodiment of the presentinvention.

FIG. 22 is a graph showing tone gradation of a recombined image, in acamera of one embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

In the following, preferred embodiments using a camera adopting thepresent invention will be described using the drawings. A camera of oneembodiment of the present invention is a digital camera, and has animaging section, with a subject image being converted to image data bythis imaging section, and the subject image then being subjected to liveview display on a display section arranged on a rear surface of a mainbody based on this converted image data. A photographer determinescomposition and photo opportunity by looking at the live view display.At the time of a release operation image data is stored in a storagemedium. Image data that has been stored in the storage medium can beplayed back and displayed on the display section if playback mode isselected.

It is also possible to divide the display section into a plurality ofregions, and creating and display a combined photograph by combining aplurality of images. A photograph of a part within this combinedphotograph may be replaced with a live view image, or an image that hasalready been taken and is stored in a storage medium. In this case, asubsidiary image (independent image constituting the combinedphotograph) that has been subjected to preprocessing is saved asindividual data, and an image of the combined photograph is made a mainimage, and an image file made up of this main image and subsidiary imageis stored (refer to S127 and S137 in FIG. 11, FIG. 20 and FIG. 21).Since the subsidiary images are stored individually, there is nodegradation in image quality even if combined photograph editing isrepeatedly edited an image of part of the combined photograph withanother image.

FIG. 1 is a block diagram mainly showing the electrical structure of acamera of one embodiment of the present invention. This camera comprisesa camera body 100 and an interchangeable lens 200 that can be attachedto and removed from the camera body. With this embodiment, thephotographing lens is an interchangeable lens, but this is not limiting,and it is also possible to have a digital camera of a type where aphotographing lens is fixed to the camera body.

The interchangeable lens 200 is made up of a photographing lens 201, adiaphragm 203, a driver 205, a microcomputer 207 and a flash memory 209,and has an interface (hereafter referred to as I/F) 300 between theinterchangeable lens 200 and the camera body 100, which will bedescribed later.

The photographing lens 201 is constituted by a plurality of opticallenses for forming a subject image, and is a fixed focal length lens ora zoom lens. The diaphragm 203 is arranged to the rear on the opticalaxis of this photographing lens 201, and the diaphragm 203 has avariable aperture to control amount of subject light flux passingthrough the photographing lens 201. Also, the photographing lens 201 iscapable of being moved in the optical axis direction by the driver 205,with focus position of the photographing lens 201 being controlled basedon control signals from the microcomputer 207, and in the case where thephotographing lens 201 is a zoom lens focal length is also controlled.The driver 205 also controls aperture of the diaphragm 203.

The microcomputer 207 that is connected to the driver 205 is connectedto the I/F 300 and the flash memory 209. The microcomputer 207 operatesin accordance with a program stored in the flash memory 209, to performcommunication with a microcomputer 121 within the camera body 100, whichwill be described later, and performs control of the interchangeablelens 200 based on control signals from the microcomputer 121.

Besides the previously described program, various information such asoptical characteristics of the interchangeable lens 200 and adjustmentvalues are stored in the flash memory 209. The I/F 300 is an interfacefor carrying out communication between the microcomputer 207 inside theinterchangeable lens 200 and the microcomputer 121 inside the camerabody 100.

Inside the camera body 100, a mechanical shutter 101 is arranged on theoptical axis of the photographing lens 201. This mechanical shutter 101controls the time for which subject light flux passes, and a well-knownfocal plane shutter or the like is adopted. Behind this mechanicalshutter 101, an image sensor 103 is arranged at a position where thesubject image is formed by the photographing lens 201.

The image sensor 103 has photodiodes that constitute each pixel arrangedtwo-dimensionally in a matrix shape, each photodiode generatesphotoelectric conversion current in accordance with received lightamount, and this photoelectric conversion current is the subject ofcharge storage by a capacitor connected to each photodiode. A Bayerarray RGB filter is arranged on the front surface of each pixel. Theimage sensor 103 is not limited to a Bayer array, and various types suchas Foveon (Registered trademark), for example, can also be used.

The image sensor 103 is connected to an analog processing section 105,and this analog processing section 105 performs wave shaping on thephotoelectric conversion signals (analog image signals) that have beenread out from the image sensor 103 after reducing reset noise etc., andalso carries out gain increase so as to achieve an appropriatebrightness.

This analog processing section 105 is connected to an A/D conversionsection 107, and this A/D conversion section 107 performs analog todigital conversion on the analog image signals, and outputs the digitalimage signals (hereafter referred to as image data) to a bus 110. Inthis application, raw image data generated before image processing inthe image processing section 109 is called RAW data.

The bus 110 is a transfer path for transferring various data, that hasbeen read out or generated inside the camera body 100, within the camerabody 100. Besides the previously described A/D conversion section 107,an image processing section 109, AE (Auto Exposure) processing section111, AF (Auto Focus) processing section 113, image compression andexpansion section 115, communication section 117, the microcomputer 121,an SDRAM (synchronous DRAM) 127, memory interface (hereafter referred toas memory I/F) 129, and display driver 133 are connected to the bus 110.

The image processing section 109 is made up of a basic image processingsection 109 a for carrying out basic image processing, a special imageprocessing section 109 b for applying special effects in the case whereart filter has been set, a combined photograph processing section 109 cfor carrying out editing processing for a combined image formed bycombining a plurality of photographs into a singe image, and a subjectdetection section 109 d for analyzing an image.

The basic image processing section 109 a performs optical black (OB)subtraction processing, white balance (WB) correction, demosaicingprocessing in the case of Bayer data, gamma/color reproductionprocessing, color matrix computation, noise reduction (NR) processing,edge enhancement processing etc. on RAW data. In a single exposure, andif art filter has not been set, image processing is completed with onlyprocessing by this basic image processing section 109 a.

Also, the special image processing section 109 b applies various specialeffects to an image that has been processed by the basic imageprocessing section, such as a pin hole effect to lower peripheralbrightness, a soft focus effect to combine an original image with animage that has been blurred, a noise effect to combine with a noiseimage, a cross filter effect to draw cross patterns on bright points, aminiature effect to blur peripheral portions etc., in accordance with aspecial effect (art filter) that has been set.

The combined photograph processing section 109 c forms a single image(combined photograph) by combining a plurality of images. In combiningimages, a template (refer to FIG. 5 which will be described later) isread out from flash memory 125, and brightness change is carried outusing gamma correction within the basic image processing section 109 aso that brightness of the images becomes the same. Also, WB is changedusing color reproduction processing within the basic image processingsection 109 a, in order to unify the overall white balance (WB). Also, aspecial effect such as a pinhole effect is applied after combinationusing the special image processing section 109 b, and correction etc. ofan incorporation image and a replacement image is carried out.

The combined photograph processing section 109 c provides a function asa first image processing section for carrying out first image processingon an image constituting the combined photograph to create a firstimage. It also functions as a second image processing section forcarrying out second image processing on the first image to create asecond image. The combined photograph processing section 109 c alsofunctions as a combined photograph processing section for creating acombined image by combining the second image in accordance with acombining position of the combined photograph.

The combined photograph processing section 109 c functioning as a firstimage processing section resizes the mage size, and rotates and trimsthe image (refer to step S123 in FIG. 11). It is also possible to onlycarry out some of these processes, and not all three, and also to haveadditional processes, as long as it constitutes basic image processingthat is carries out before combined photograph creation. The combinedphotograph processing section 109 c functioning as a second imageprocessing section carries out color correction and brightnesscorrection for the images (refer to steps S129 and S131 in FIG. 11). Itis not necessary to carry out both of these processes, and there may beadditional processing, for example image quality adjustment etc. carriedout after basic image processing.

The subject detection section 109 d detects a subject such as a person'sface or a pet, by analysis of an image using pattern matching etc. Inthe event that it is possible to detect a subject, information such asthe type, size, position and reliability etc. of the subject within animage is acquired. The subject detection section 109 d and the combinedphotograph processing section 109 c act cooperatively to function as ananalysis section to analyze feature sections of the image and outputresults of analysis (refer to S125 in FIG. 11). As feature sections ofan image there are at least one of a histogram, subject detectionresults, and size, position and color characteristics of a main subject.

The AE processing section 111 measures subject brightness based on imagedata that has been input via the bus 110, and outputs this subjectbrightness information to the microcomputer 121 via the bus 110. Adedicated photometric sensor for subject brightness measurement may beprovided, but in this embodiment subject brightness is calculated basedon image data.

The AF processing section 113 extracts signals for high frequencycomponents from image data, acquires a focus evaluation value usingintegration processing, and outputs the focus evaluation value via thebus 110 to the microcomputer 121. In this embodiment, focusing of thephotographing lens 201 is carried out using the so-called contrastmethod. With this embodiment AF control using a contrast method is givenas an example, but it is also possible to carry out focusing by AFcontrol using phase difference AF, where a phase difference sensor isprovided on the optical path, or a phase difference sensor is providedon an image sensor.

At the time of storage of image data to the storage medium 131, theimage compression and expansion section 115 subjects image data that hasbeen read out from the SDRAM 127 to compression in accordance withvarious compression formats such as JPEG compression in the case of astill picture or MPEG in the case of a movie. The microcomputer 121attaches a header necessary for configuring a JPEG file or an MPO (MultiPicture Object) file or MPEG file to the JPEG image data or MPEG imagedata, to create a JPEG file or an MPO file or MPEG file, and stores thecreated file in the storage medium 131 via the memory I/F 129. With thisembodiment, an image file made up of an image of a combined photograph(main image) and images of individual combined photographs (subsidiaryimages) is stored as an MPO file. The file structure will be describedlater in FIG. 20 and FIG. 21.

The image compression and expansion section 115 also carries outexpansion of JPEG image data and MPEG image data for image playbackdisplay. In the expansion, a file that is stored in the storage medium131 is read out, and after being subjected to expansion processing inthe image compression and expansion section 115 the expanded image datais temporarily stored in the SDRAM 127. With this embodiment, the JPEGcompression system and MPEG compression system are adopted as the imagecompression system, but the compression system is not limited to thisand another compression systems may be used, such as TIFF, H.264 etc.

The communication section 117 performs communication with externalsections by means of wire communication using USB or LAN etc., orwireless communication. Templates for division of combined photographsthat are stored within the flash memory 125 are updated or added bymeans of the communication section 117.

The microcomputer 121 provides a function as a control section for thisentire camera, and performs overall control of various sequences of thecamera. Besides the previously described I/F 300, an operation section123 and a flash memory 125 are connected to the microcomputer 121.

After the first image processing has been carried out, the microcomputer121 functions as a control section to create a combined photograph bycombining the second image that has been subjected to second imageprocessing in accordance with combining position of the combinedphotograph using the combined photograph processing section 109 c, andstores the combined photograph and the first image as a single imagefile. This function is executed in accordance with the flowchart shownin FIG. 11, for example.

The operation section 123 includes operation members such as variousinput buttons, like a power supply button, release button, a moviebutton, playback button, menu button, cross-shaped key, OK button,delete button, magnification button and various input keys, and detectsoperating states of these operation members and outputs the result ofdetection to the microcomputer 121. The microcomputer 121 executesvarious sequences according to user operation based on the result ofdetection of the operation members from the operation section 123. Thepower supply button is an operation member for instructing to turn apower supply of the digital camera on or off. If the power supply buttonis pressed, the power supply of the digital camera is turned on, and ifthe power supply button is pressed once again the power supply of thedigital camera is turned off.

The release button is made up of a first release switch that turns onwhen the button is pressed down half way, and a second release switchthat is turned on when the button is pressed down further from thehalf-pressed state to be pressed down fully. The microcomputer 121executes shooting preparation sequences such as an AE operation and AFoperation if the first release switch is turned on. Also, if the secondrelease switch is turned on shooting is carried out by executing aseries of shooting sequences to control the mechanical shutter 101 etc.,acquire image data based on a subject image from the image sensor 103etc., and store this image data in the storage medium 131.

The movie button is an operation button for designating start or finishof movie shooting, and if the movie button is initially operated movieshooting commences, and movie shooting finishes when it is operatedagain. The playback button is an operation button for setting andcanceling playback mode settings, and if playback mode is set image dataof a taken image is read out from the storage medium 131, and a takenimage is playback displayed on the display panel 135. The delete buttonis an operation button for designating and deleting an image in casessuch as where playback images are displayed as a list, or a combinedphotograph is being displayed, for example. The magnification button isan operation button for enlarging a display image at the time ofplayback display.

The menu button is an operation button for causing display of menuscreens on the display panel 135. It is possible to carry out thevarious camera settings on the menu screens. As camera settings thereare shooting mode settings such as normal shooting mode and combinedphotograph for example, and finish settings such as natural, vivid,flat, portrayed, art filter etc. As art filters there are pop art, toyphoto, fantastic focus, rough monochrome, Diorama, Crystal etc. It isalso possible to have various settings on the menu screens, such asstyle of a combined photograph, selection of the replacement image for acombined photograph, and whether or not an image before editing will bestored at the time of editing a combined photograph. In the case of acombined photograph, display of a live view (LV) image is updated inaccordance with a filter and incorporation image selection state (referto S47 in FIG. 7).

A touch input section 123 a is also provided on the operation section123. The display panel 135 is capable of touch operation, and the touchinput section 123 a detects a position that has been touched by a useretc. and outputs to the microcomputer 121.

The flash memory 125 stores a program for executing the varioussequences of the microcomputer 121. The microcomputer 121 carries outoverall control of the camera based on this program.

The SDRAM 127 is an electrically rewritable volatile memory fortemporary storage of image data etc. This SDRAM 127 temporarily storesimage data that has been output from the A/D conversion section 107, andimage data that has been processed in the image processing section 109and image compression and expansion section 115 etc.

The memory I/F 129 is connected to the storage medium 131, and carriesout control for reading and writing of data, such as image data andheaders attached to image data, to and from the storage medium 131. Thestorage medium 131 functions as a storage section for storing (S137 ofFIG. 11) the first image (refer to S123 of FIG. 11, and subsidiaryimages P11-P14 of FIG. 20, and FIG. 21), and the images of the combinedphotograph (refer to S133 of FIG. 11, and the main images P1, P2 of FIG.20 and FIG. 21). The storage medium 131 is a storage medium such as amemory card that can be loaded into and taken out of the camera body100, but this is not limiting and it may also be a hard disk or the likebuilt into the camera body 100.

The LCD driver 133 is connected to the display panel 135, and reads outfrom the SDRAM 127 and storage medium 131 to display an image on thedisplay panel 135 based on image data that has been expanded by theimage compression and expansion section 115. The display panel 135 isarranged on a rear surface of the camera body 100, and carries out imagedisplay. As image display modes, there are Quickview Display fordisplaying stored image data for only a short time, playback display ofimage files for still images and movies that have been stored in thestorage medium 131, and movie display such as live view display. As adisplay section it is possible to adopt various display panels such as aliquid crystal display panel (LCD), or organic EL etc.

Next, editing operations for a combined photograph of the camera of thisembodiment will be described using FIG. 2 to FIG. 5. FIG. 2 shows liveview display for the case where combined photograph mode is selected,and a screen of the display panel 135 has been divided into 4. Diagonallines within a divided screen (double lines close to the elbows of aperson and double lines behind the legs) indicate that a subject ismoving. The appearance of the subject is shown on the left side of FIG.2, while the right side shows a display state of the display panel 135.

First, as shown in 135 a of FIG. 2, within the 4 divided screens liveview display is performed on the upper left screen. If the user touchesthe upper left of the display panel 135 when they wish to acquire thisas an image for the combined photograph, then as shown in 135 b shootingis decided and live view display is carried out on the next screen(upper right screen). Next, 135 c is the appearance of deciding shootingfor the upper right screen, while 135 d shows the appearance of decidingshooting for the lower left screen. If shooting of the lower rightscreen is decided (refer to 135 e), shooting is completed and a combinedphotograph is possible.

With the example shown in FIG. 2, shooting is decided by touching thedisplay panel 135, and live view display is performed in order at theupper left→upper right→lower left→lower right. However, live viewdisplay does not have to be carried out in this order. For example, asshown in FIG. 3, it is also possible to carry out live view display suchthat when live view display is being performed on the upper left screen(refer to 135 f in FIG. 3), if the lower left screen which is notperforming live view display is touched, live view display is thenperformed on this screen (refer to 135 g in FIG. 3). The actualoperation of updating this shooting frame will be described later usingsteps S161 and S163 in FIG. 14.

Also, in FIG. 2 shooting is decided if the display panel 135 is touchedduring live view display, but even with a screen that has been decided,the live view display screen can be returned to if the display panel 135is touched again. With the display panel 135 h shown in FIG. 3, shootinghas been decided on 3 screens (upper left, upper right, and lower leftscreens), but if the lower left screen for which shooting has beendecided is touched, the lower left screen of the display panel 135 jbecomes a live view display screen.

Also, with this embodiment, even after shooting has been decided for allframes, and a combined photograph has been formed, it is possible toreplace some of the images with another image. For example, FIG. 4 showsa situation where four images have been decided, but the user hasspecified to replace the image on the lower right screen with anotherimage. With this embodiment, the frame of the image is displayed bytouching, to indicate that it is an image to be recombined.

With examples of combined photographs shown in FIG. 2 to FIG. 4, theyhave been divided into 4, namely upper left, upper right, lower left,and lower right, but the division of the screen is not limited to this.For example, it is also possible to divide into various formats, asshown in FIG. 5. The format of this division (template, style) is storedin the flash memory 125, so that the user can set according to taste.

Next, main processing of the camera of this embodiment will be describedusing the flowcharts shown in FIG. 6 and FIG. 7. The flowcharts shown inFIG. 6-FIG. 11, FIG. 13-FIG. 15, and FIG. 17-FIG. 19 are executed by themicrocomputer 121 controlling each section in accordance with programsstored in the flash memory 125.

If the power supply button within the operation section 123 is operatedand the power supply is turned on, operation of the main flow shown inFIG. 6 is started. Once operation has commenced, initialization is firstexecuted (S1). As part of the initialization, mechanical initializationand electrical initialization, such as initialization of various flagsetc., are carried out. As one of the various flags, a storage inprogress flag indicating whether or not movie storage is in progress isreset to off (refer to Steps S13 and S15).

Once initialization has been carried out, it is next determined whetheror not the playback button has been pressed (S3). Here, determination iscarried out by detecting the operating state of the playback buttonwithin the operation section 123. If the result of this determination isthat the playback button has been pressed, playback/editing mode isexecuted (S5). Here, image data is read out from the storage medium 131,and a table of still images and movies is displayed on the LCD 135. Theuser selects an image from within the table by operating thecross-shaped key and decides on the image using the OK button. Editingof a combined photograph is also carried out in this step. Detailedoperation of this playback/editing will be described later using FIG.17.

If playback/editing is executed in step S5, or if the result ofdetermination in step S3 was that the playback button had not beenpressed, it is determined whether or not camera setting will be carriedout (S7). When the menu button within the operation section 123 has beenoperated, camera setting is carried out on a menu screen. In this steptherefore, determination is based on whether or not this camera settinghas been carried out.

If the result of determination in step S7 is camera setting, camerasetting is carried out (S9). As described previously, it is possible tocarry out various camera settings on the menu screens. As camerasettings there are, as described previously, shooting mode settings suchas normal shooting mode and combined photograph, for example, and finishsettings such as natural, vivid, flat, portrait, art filter etc. As artfilters there are pop art, toy photo, fantastic focus, rough monochrome,Diorama, Crystal etc. Various settings are also possible such as styleof a combined photograph, selection of a replacement image for acombined photograph, whether or not an image before editing will bestored at the time of combined photograph editing, selection of an imagefor embedding an already stored image in a specified frame of a combinedphotograph, etc.

If camera setting has been carried out in step S9, or if the result ofdetermination in step S7 was not camera setting, it is next determinedwhether or not the movie button has been pressed (S11). Here themicrocomputer 121 carries out determination based on operating state ofthe movie button input from the operation section 123.

If the result of determination in step S11 is that the movie button hasbeen pressed, the storage in progress flag is determined (S13). Thestorage in progress flag is set to on (1) if movie storage is progress,or set to off (0) if a movie is not being shot. In this step the flag isinverted, that is, if it is set to on (1) it is inverted to off (0), andif it is set to off (0) it is inverted to on (1).

Once inversion of the storage in progress flag has been carried out instep S13, it is next determined whether or not storage of a movie is inprogress (S15). Here, determination is based on whether the storage inprogress flag, which was inverted in step S13, is set to on or set tooff.

If the result of determination in step S15 is that movie storage is inprogress, a movie file is created (S19). Movie storage is carried out instep S61, which will be described later, but in this step a movie filefor movie storage is created, and preparation so as to be able to storeimage data of the movie is performed.

On the other hand, if the result of determination is that move storageis not in progress, the movie file is closed (S17). The movie file isclosed in this step because the movie button has been pressed and movieshooting is completed. When closing the movie file, the movie file isplaced in a state of being capable of being played back by storing anumber of frames in a header of the movie file etc., and file writing iscompleted.

Once the movie file has been closed in step S17, or if a movie file iscreated in step S19, or if the result of determination in step S11 wasthat the movie button was not pressed, it is next determined whether ornot the operation section has been operated in combined photograph mode(S21). As described previously, setting of combined photograph mode ispossible in the camera settings of step S9. In this step it isdetermined whether or not an operating section of the operation section123 was operated in a state where this combined photograph mode is set.

If the result of determination in step S21 is that an operating sectionhas been operated in combined photograph mode, a combined photographoperation is executed (S23). With this combined photograph operation,various operations for carrying out editing of the combined photograph,for example, shooting frame update, cancel operation, restorationoperation, temporary save operation, temporary save and write operation,etc. are carried out. Detailed operation of this combined photographoperation will be described later using FIG. 14 and FIG. 15.

If playback/editing has been executed in step S23, or if the result ofdetermination in step S21 was that an operating section was not operatedin combined photograph mode, it is determined whether or not the releasebutton has been pressed down halfway, in other words, whether or not thefirst release switch and gone from off to on (S31). For thisdetermination, the state of the first release switch that is linked tothe release button is detected by the operation section 123, anddetermination is carried out based on the result of detection. If theresult of detection is that the first release switch has changed fromoff to on, the result of determination becomes Yes, while on the otherhand if an on state or an off state is maintained the result ofdetermination becomes No.

If the result of determination in step S31 is that the release buttonhas been pressed down half way, then in the case where the first releasehas transitioned from off an AE/AF operation is executed (S33). Here,the AE processing section 111 detects subject brightness based on imagedata acquired by the image sensor 103, and calculates shutter speed andaperture value etc. for correct exposure based on this subjectbrightness.

Also, an AF operation is carried out in step S33. Here, the driver 205moves focus position of the photographing lens 201 by means of themicrocomputer 207 inside the interchangeable lens 200, so that a focusevaluation value that has been acquired by the AF processing section 113becomes a peak value. As a result, if the release button is pressed downhalfway when movie shooting is not being carried out, focusing of thephotographing lens 201 is carried out at that point in time. After thatprocessing advances to step S35.

If the result of determination in step S31 is that the release buttonhas not transitioned from off to the first release, it is nextdetermined whether or not the release button has been pressed downfully, and the second release switch has been turned on (S37). In thisstep, the state of the second release switch that is linked to therelease button is detected by the operation section 123, anddetermination is carried out based on the result of detection.

If the result of determination in step S37 is that the release button ispressed down fully and that the second release which has been turned on,still image shooting is carried out using the mechanical shutter 101(S39). Here, the diaphragm 203 is controlled with the aperture valuethat was calculated in step S33, and the shutter speed of the mechanicalshutter 101 is controlled with the calculated shutter speed. Then, oncethe exposure time corresponding to the shutter speed has elapsed, imagesignals are read from the image sensor 103, and RAW data that has beenprocessed by the analog processing section 105 and the A/D conversionsection 107 is output to the bus 110.

Once shooting using the mechanical shutter 101 has been carried out,image processing is carried out (S41). Here, RAW data that has beenacquired by the image sensor 103 as a result of shooting using themechanical shutter 101 is read out, and image processing is carried outby the image processing section 109. Detailed operation of this imageprocessing will be described later using FIG. 8.

Once image processing has been carried out, still picture storage isnext carried out (S43). Here, image data for a still picture to whichimage processing has been applied is stored in the storage medium 131.In the case of having shot with a combined photograph, then in the eventthat it is desired to cancel (delete) part of an image within thecombined photograph, it is possible to delete using a cancel operation.Detailed operation of this still picture storage will be described laterusing FIG. 13.

Once still picture storage has been carried out, it is next determinedwhether or not combined photograph mode has been set (S45). As describedpreviously, combined photograph mode is set on a menu screen etc., andin this step whether or not this combined photograph mode has been setis determined.

If the result of determination in step S45 is that combined photographmode has been set, update to live view (LV) display is carried out(S47). If combined photograph mode has been set, live view display isupdated in accordance with a style (template, refer to FIG. 5) and anumber of images. In a case where live view display is carried out allthe time for all frames, live view display is carried out at parts otherthan where there are already taken images. Also, in the case of liveview display one frame at a time (refer to the display method of FIG.2), live view display is switched from a taken frame to the next frame(for example, in display panels 135 a-135 e of FIG. 2, in the orderupper left→upper right→lower left→lower right). In either case, takenframes display taken images. It is also possible to display so as toknow which are already taken images and which is live view display,using OSD (on-screen display), and it is also possible for already takenimages to be displayed so as to know the order in which they were taken.After that processing advances to step S35.

If the result of determination in step S37 was not release operation2nd, that is in the case where the second release switch is not on, AEis next carried out (S51). In the case where S37 is passed through frompreviously described step S31, it is a situation where no operation wascarried out for the release button, and in this case live view displayof step S57, which will described later, is carried out. In step S51,exposure control is carried out in order to carry out live view display.Specifically, shutter speed and ISO sensitivity for the electronicshutter of the image sensor 103 in order to carry out live view displayat the correct exposure are calculated.

Once AE has been carried out, shooting using the electronic shutter iscarried out (S53). Here image data is acquired using the electronicshutter of the image sensor 103. Next, image processing is carried outon the acquired image data (S55). The image processing here is for liveview display, and in the case where a special effect has been set, imageprocessing in accordance with the set special effect is also carriedout. Detailed operation of this image processing will be described laterusing FIG. 8.

If image processing has been carried out, live view display on thedisplay panel 135 is carried out (S57). Image data is acquired in stepS53, and image processing is carried out in step S55, and so in thisstep update of the live view display is carried out. A photographer candetermine composition and photo opportunity by looking at the live viewdisplay.

Once live view display has been carried out in step S57, it isdetermined whether or not movie storage is in progress (S59). Here it isdetermined whether or not the storage in progress flag is on. If theresult of this determination is that movie storage is in progress, moviestorage is carried out (S61). Here, image data read out from the imagesensor 103 is subjected to image processing to give image data for amovie, and stored in a movie file. After that processing advances tostep S35.

In step S35 it is determined whether or not to power off. In this stepit is determined whether or not the power supply button of the operationsection 123 has been pressed again. If the result of this determinationis not power off, processing returns to step S3. On the other hand, ifthe result of determination is power off, a termination operation of themain flow is executed and then the main flow is terminated.

In this way, with the main flow of the one embodiment of the presentinvention, in the case where combined photograph mode is setplayback/editing is possible (S23), shooting is carried out if therelease button is pressed down fully (release operation second), andupdate of live view display is performed.

Next, the image processing of steps S41 and S55 will be described usingthe flowchart shown in FIG. 8. If the image processing flow is entered,basic image processing is first carried out (S71). Here, the basic imageprocessing section 109 a applies optical black (OB) subtraction, whitebalance (WB) correction, demosaicing processing, color reproductionprocessing, brightness change processing, edge enhancement processing,noise removal (NR) processing etc. to image data that has been read outfrom the image sensor 103 and subjected to AD conversion by the A/Dconversion section 107. Detailed operation of this basic imageprocessing will be described later using FIG. 9.

If basic image processing has been carried out, it is next determinedwhether or not art filter has been set (S73). An art filter is set inthe previously described camera setting (S9), and so in this step it isdetermined whether or not an art filter has been set.

If the result of determination in step S73 is that an art filter hasbeen set, special image processing is carried out (S75). Here, specialimage processing is applied in accordance with the type of art filterthat has been set. Detailed operation of this special image processingwill be described later using FIG. 10.

If special image processing has been carried out, or if the result ofdetermination in step S73 is that an art filter has not been set, it isnext determined whether or not it is a combined photograph (S77). If theresult of this determination is that combined photograph mode has beenset, combined photograph creation is carried out (S79). With thiscombined photograph creation, preprocessing such as resizing androtation is carried out, and after carrying out color change andbrightness change image combination is carried out in accordance with astyle (template), to create a combined photograph. Detailed operation ofthis combined photograph creation will be described later using FIG. 11.

If combined photograph creation is carried out, or if the result ofdetermination in step S77 is that combined photograph mode has been set,the originating processing flow is returned to.

In this way, with the image processing of this embodiment, aftercarrying out basic image processing and carrying out special imageprocessing as required, combined photograph creation is carried out ifcombined photograph mode has been set.

Next, detailed operation of the basic image processing in step S71 (FIG.8) will be described using the flowchart shown in FIG. 9. If the basicimage processing flow is entered, first optical black (OB) calculationis carried out (S81). In this step, an optical black value, which isascribable to dark current etc. of the image sensor 103, is respectivelysubtracted from a pixel value of each pixel constituting the image databy an OB calculation section within the basic image processing section109 a.

Once OB calculation has been carried out, white balance (WB) correctionis carried out (S83). In this step WB correction is carried out on imagedata by the WB correction section within the basic image processingsection 109 a, in accordance with a white balance mode that has been setin advance. Specifically, correction is carried out by reading out Rgain and B gain corresponding to the white balance mode set by the userfrom the flash memory 125 of the camera body, and multiplying the imagedata of the Bayer array by these values. Alternatively, in the case ofauto white balance, R gain and B gain are calculated from RAW data, andcorrection is carried out using these values.

Next, demosaicing processing is carried out (S85). In this step, imagedata that has been subjected to white balance correction is converted todata having each pixel made up of RGB data by a demosaicing processingsection within the basic image processing section 109 a. Specifically,data not at that pixel is obtained by interpolation from around thatpixel position, and converted to RGB data. However, in the case whereRAW data is already in the same format as RGB data (such as RAW data setthat has been acquired from a Foveon sensor), demosaicing processingdoes not need to be carried out.

Once demosaicing processing has been carried out, color reproductionprocessing is carried out (S87). In this step, a linear transformationto multiply the image data by a color matrix coefficient according tothe set white balance mode is carried out by a color reproductionprocessing section within the basic image processing section 109 a, tocorrect the color of the image data. The color matrix coefficient isstored in the flash memory 125, and so it is read out and used.

Once color reproduction processing has been carried out, brightnesschange processing is carried out (S89). In this step a gamma tablestored in the flash memory 125 is read out by a gamma processing sectionwithin the basic image processing section 109 a, and gamma correctionprocessing is carried out on the image data. After gamma conversion withRGB and color conversion to YCbCr, further gamma conversion is carriedout with Y.

If gamma conversion has been carried out, edge enhancement is carriedout next (S91). In this step, an edge enhancement processing sectionwithin the basic image processing section 109 a enhances edges of theimage data, by extracting edge components, for image data that has beensubjected to gamma correction, using a band pass filter, multiplying bya coefficient corresponding to a degree of edge enhancement, and addingto the image data.

Next, NR (noise removal) is carried out (S93). In this step an image issubjected to frequency analysis, and processing is carried out to reducenoise in accordance with frequency. Once noise removal processing hasbeen carried out, the originating processing flow is returned to.

Next, detailed operation of the special image processing in step S75(FIG. 8) will be described using the flowchart shown in FIG. 10.

If the flow for special image processing is entered, it is firstdetermined whether or not toy photo has been set as an art filter mode(S101). If the result of this determination is that toy photo has beenset, a shading effect is added (S103). Here, the special imageprocessing section 109 b applies a shading effect to the periphery ofthe image by creating a gain map (gain values are one or less), suchthat brightness is gradually lowered in accordance with distance fromthe center of the image, and multiplying each pixel by a gaincorresponding to the pixel.

If shading addition has been carried out, or if the result ofdetermination in step S101 was not toy photo, it is determined whetheror not fantastic focus has been set as the art filter mode (S105). Ifthe result of this determination is that fantastic focus has been set,soft focus processing is carried out (S107). Here, the special imageprocessing section 109 b blurs an original image, and combines thisblurred image and an image before blurring at a specified ratio (forexample, a ratio of blurred image:image before blurring=2:3).

If soft focus processing has been carried out, or if the result ofdetermination in step S105 was not fantastic focus, it is nextdetermined whether or not rough monochrome has been set as the artfilter mode (S109). If the result of this determination is that therough monochrome has been set, noise superimposing processing is carriedout (S111). Here, noise addition processing is applied to the image databy the special image processing section 109 b. Noise addition processingis processing to add a previously created noise pattern image to thesubject image. It is also possible to create a noise pattern image basedon a random number etc.

If noise superimposing processing has been carried out, or if the resultof determination in step S109 was not rough monochrome, it is nextdetermined whether or not Diorama has been set as the art filter mode(S113). If the result of this determination is that Diorama has beenset, blurring processing is carried out (S115). Here, the special imageprocessing section 109 b carries out gradual blurring processing of theoriginal image above and below, or to the left and right, or inaccordance with distance from, a center of an AF target.

If blurring processing has been carried out, or if the result ofdetermination in step S113 was not Diorama, it is next determinedwhether or not Crystal has been set as the art filter mode (S117). Ifthe result of this determination is that Crystal has been set, crossfilter processing is carried out (S119). Here, the special imageprocessing section 109 b detects a bright point from within an image,and with this bright point as a center draws a cross shape or crosspattern, which is an effect for creating a shiny effect on a pluralityof radial shapes.

If cross filter processing has been carried out, or if the result ofdetermination in step S117 was that crystal was not set, the flow forspecial image processing is terminated, and the originating flow isreturned to.

Next, detailed operation of the combined photograph creation in step S79(FIG. 8) will be described using the flowchart shown in FIG. 11. As wasdescribed previously, with the flow for image processing shown in FIG.8, first of all basic image processing is carried out, then specialimage processing is carried out, and next this combined photographcreation is carried out. With this combined photograph creation, roughlysummarized, images are combined in a style (template) such as is shownin FIG. 5. In carrying out this combination, if there are frames thathave not yet been subjected to the preprocessing outlined instepsS121-S127, preprocessing is applied. If the preprocessing has beenapplied, color change and brightness change etc. are carried out, imageprocessing to embed images into the style is executed, and then imagestorage is carried out.

If the flow for combined photograph creation is entered, it is firstdetermined whether or not there are frames to which preprocessing hasnot been applied (S121). Here, for images of all frames to be combined,it is determined whether or not preprocessing such as resizing,rotating, trimming, image analysis etc., for which it is not necessaryto execute processing from step S128 and after each time, has beencompleted.

If the result of determination in step S121 was that there are framesthat have not yet been subjected to preprocessing, resizing, rotationand trimming are carried out (S123). Here, in order to embed into astyle such as is shown in FIG. 5, the combined photograph processingsection 109 c carries out processing such as resizing, rotation andtrimming according to the frame.

Once resizing etc. has been carried out, image analysis is nextperformed (S125). An original image (replacement image) to be combinedis analyzed, and image features (for example brightness distribution,color distribution etc.) are acquired. It is also possible to performanalysis using an image for which image processing such as basic imageprocessing has already been completed, or to perform analysis on rawdata constituting the original image.

Once image analysis has been carried out, frame image storage is carriedout next (S127). Here, images that have been subjected to thepreprocessing of steps S123 and S125 are stored in the SDRAM 127. FIG.16A shows incorporation image storage regions for display and storage inthe SDRAM 127, with the region R1 being a region for number of framesincorporated into the combined photograph, and region R2 being a saferegion for number of frames incorporated into the combined photograph.In this step S127, image data for images that have been subjected topreprocessing and the results of image analysis are stored incorresponding regions of the frame image region R1 (in any of frameimage region 1-frame image region n).

Once frame image storage has been carried out, or if the result ofdetermination in step S121 was that there were no frames that have notyet been subjected to preprocessing, color change is next carried out(S129). Here, the combined photograph processing section 109 c correctsCbCr so that images to be combined become substantially the same color.At this time processing is carried out using data for each frame imageregion, but the processing does not update data within a frame imageregion R1 of the SDRAM 127. This is in order to save data later in anMPO format. This MPO format data save will be described later using FIG.20 and FIG. 21.

As a color change correction method, for example, CbCr of an image isoffset so that a peak of color distribution (distribution on the CbCrplane) for all images to be combined becomes an average value of eachpeak. This correction method will be described using FIG. 12B and FIG.12C. The graph (b-1) in FIG. 12B shows color difference (Cb) for image 1and image 2, while the graph (c-1) in FIG. 12C shows color difference(Cr). If color difference is offset in image 1 and image 2 respectivelyso that color distribution peak becomes an average value (refer to graph(b-2) in FIG. 12B and graph (c-2) in FIG. 12C), the color difference forboth images becomes substantially the same (refer to graph (b-3) in FIG.12B and graph (c-3) in FIG. 12C).

Once color change has been carried out in step S129, brightness changeis next carried out (S131). Here, correction is performed so that imagesto be combined by the combined photograph processing section 109 cbecome of the same brightness. As a correction method, for example,correction is performed so that average values for each image aresubstantially matched to an average value for brightness distribution ofall images to be combined. At this time if conversion is carried outwith brightness gamma (gamma for only a luminance component), colorwould appear unnatural in the case where brightness was changedsignificantly. It is therefore possible, for example, to perform gammacorrection with RGB color space after RGB conversion.

An example of brightness change will be described using FIG. 12A. Thegraph (a-1) of FIG. 12A shows brightness of image 1 and image 2. Theyare both subjected to table conversion in RGB color space (refer tograph (a-2)). As will be understood from (a-1), image 2 has a lot of lowbrightness components overall, and the low brightness side is raised upas a result of table conversion. After conversion, image 1 and image 1have substantially the same brightness distribution, as shown in graph(a-3).

Once brightness change has been carried out in step S131, combination isnext carried out (S133). Here, the combined photograph processingsection 109 c combines each image on a background image. Specifically, acombined photograph is created by carrying out combination by embeddingimages at each position of a style such as is shown in FIG. 5.

If combination has been carried out a special effect is applied next(S135). Here, a special effect such as shading or blurring is added toan image that has been combined. A special effect was added to eachimage in step 75 of FIG. 8, and in this step a special effect is addedto a combined image.

Once a special effect has been added, image storage is next carried out(S137). Here, a combined effect is stored in a combined image storageregion for display and storage. Once image storage has been carried outthe originating flow is returned to.

In this way, in the flow for combined photograph creation, in the casewhere preprocessing has not yet been applied, processing such asresizing and image analysis are applied (S123, S125), this preprocessedimage data is stored in a frame image region R1 of the SDRAM 127 (S127),and after color change and brightness change have been carried out forimages that have been preprocessed (S129, S131), images for a combinedphotograph are combined (S133), and image storage is carried out (S137).Image data before carrying out preprocessing, color change andbrightness change is stored in the frame image region R1, which meansthat it is possible to read out image data at the time of preprocessingwhen replacing with a replacement image after combined photographcreation, and it is possible to carry out appropriate color change andbrightness change.

Also, in the flow for combined photograph creation, images that havebeen subjected to preprocessing, such as resizing rotation and trimming,and image analysis results are stored as frame images (subsidiaryimages) (refer to S123-S127). Also, combination processing is appliedusing images that have been subjected to color change and brightnesschange, to create an image for a combined photograph (main image), whichis then subjected to image storage (S129-S137). Since individualsubsidiary images constituting the combined photograph are storedindividually, there is no degradation in the images of the combinedphotograph even if part of an image constituting the combined photographis repeatedly replaced.

Accordingly, in the flow for combined photograph creation, aftercreating the combined photograph and storing this combined photographand subsidiary images (first images) as a single image file, it ispossible to create a new combined photograph by replacing parts of theimage constituting the combined photograph with replacement images thatare different to the subsidiary images (first images). In this case,replacement images are subjected to image processing by the first imageprocessing section (namely step S123), then subjected to further imageprocessing by the second image processing section (namely steps S129 andS131), and then a new combined photograph is created by the combinedphotograph processing section using the replacement images that havebeen subjected to this processing, and second images resulting fromsubjecting subsidiary images (first images) other than the replacementimages stored in the image file to image processing by the second imageprocessing section (steps S129 and S131). Creation of this combinedphotograph will be described later using FIG. 20 and FIG. 21.

Next, detailed operation of the still picture storage in step S43 (FIG.7) will be described using the flowchart shown in FIG. 13. This stillpicture storage performs processing in order to shoot an image using thepreviously described mechanical shutter 101 (S39), subsequently executeimage processing (S43), and then store the image data that has beensubjected to image processing in the storage medium 131.

If the flow for still pictures storage is entered, it is firstdetermined whether or not it is combined photograph mode (S141). If theresult of this determination is that it is not combined photograph mode,it is normal mode, and Quickview Display is carried out (S157). Here,images stored in the storage medium 131 are temporarily displayed for aspecified time on the display panel 135. In this way it is possible forthe user to confirm taken images.

Once Quickview Display has been carried out, still picture storage isnext carried out (S159). Besides a combined photograph, images that havebeen subjected to image processing are subjected to JPEG compression andstored in the storage medium 131. However, this is not limited and it isalso possible to store in an uncompressed format (for example TIFFetc.), or to use another compression format. It is also possible tostore RAW images.

If the result of determination in step S141 and is that it is combinedphotograph mode, it is determined whether or not to perform taken imagestorage (S143). With combined photograph mode, even in a case ofembedding each image in a style such as is shown in FIG. 5, but wherecombined photograph mode has been set, images that are not embeddedwithin the style can also be stored. In this step, it is determinedwhether or not the user has turned on a setting for whether or not tostore each frame of the combined photograph, in the camera setting ofstep S9 in FIG. 6.

If the result of determination in step S143 is taken image storage,still picture storage is carried out (S145). Here, similarly to stepS159, a picture is taken as a single taken image, and not a combinedphotograph, subjected to compression etc. using JPEG or the like, andstored in the storage medium 131.

Once still picture storage has been carried out in step S145, or if theresult of determination in step S143 was not taken image storage, it isnext determined whether or not combining is complete (S147). Here, anumber of images that satisfies a style that has been set in advance aretaken, and it is determined whether or not they have been combined. Forexample, with the combined photograph that was described using FIG. 2and FIG. 3, four photographs are taken and it is determined whether ornot they have been combined. Also, for example, in a case of combiningfour images, it is also determined that incorporation is complete in acase where two images are taken, and two images are selected from storedimages.

If the result of determination in step S147 was incorporation complete,Quickview Display is carried out (S149). Here, after completion of allimages as a combined photograph, this combined photograph is subjectedto Quickview Display on the display panel 135. During Quickview Display,similarly to the flow for playback, which will be described later,magnified display is also possible.

Once Quickview Display has been carried out, it is next determinedwhether or not there has been a cancel operation (S151). There may thesesituations where the user is considering deleting some photographs andreplacing them, when looking at the Quickview Display. Therefore, withthis embodiment, in the case where Quickview Display is carried out anda cancel operation is performed within a specified time (for example,three seconds), a designated frame is canceled, and shooting can becarried out again. As a cancel operation, it is possible to performdesignation on a touch panel, or to cancel using a delete button.

If the result of determination in step S151 is that the cancel operationhas been carried out, a combined photograph operation is carried out(S155). Within a subroutine for the combined photograph operation, acancel operation is executed (refer to S165-S173 in FIG. 14, which willbe described later).

On the other hand, if the result of determination in step S151 is that acancel operation has not been performed, combined photograph storage iscarried out (S153). Here, combined image data for a completed combinedphotograph is stored in the storage medium 131.

If combined photograph storage was carried out in step S153, or if acombined photograph operation was carried out in step S155, or if theresult of determination in step S147 was not incorporation complete, orif still picture storage was carried out in step S159, the originatingprocessing flow is returned to.

In this way, in the still picture storage flow, combined photographstorage is not carried out until a number of images determined by thestyle have been assembled (S147→No), and storage of a combinedphotograph is carried out once the number of images have been assembled(S153). Also, even after completion of a combined photograph it ispossible to cancel an image of the combined photograph that has beendesignated by carrying out a cancel operation (S151).

Next, detailed operation of the combined photograph operation of stepS23 (FIG. 6) and step S155 (FIG. 13) will be described using theflowcharts shown in FIG. 14 and FIG. 15. As described previously, thiscombined photograph operation is executed when various operations arecarried out in order to carry out editing of the combined photograph.

If the flow for combined photograph operation is entered, it is firstdetermined whether or not a taken frame update operation has beencarried out (S161), and if the result of this determination is that ataken frame update operation has been performed update of a taken frameis carried out (S163). As was described for the display panels 135 f,135 g of FIG. 3, which frame a taken image is embedded within in acombined photograph style can be changed as appropriate. In the casewhere an operation to update a frame that will be taken has beenperformed by operation of a cross-shaped key or okay button or a touchpanel operation etc., a frame to be taken is updated. At this time, inorder to make known which frame is being taken, it is possible todisplay a border on the frame, display an icon, display frames otherthan the taken frame darkly, or reduce the chroma of frames other thanthe taken frame. In the case where a frame is touched during live viewdisplay, a release operation (first release and second release, or firstrelease only) is executed. In the case where live view display is beingcarried out with the plurality of frames, a frame to be taken isselected using the release button.

If update of a taken frame has been carried out in step S163, or if theresult of determination in step S161 was that an update operation for ataken frame was not carried out, it is next determined whether or not acancel operation was performed (S165). As was described for the flow forstill picture storage, there are situations where some images making upa completed combined photograph are deleted, or it is desired to replacean image. With this embodiment, therefore, a frame that has been takenis canceled in cases such as when a taken frame is selected using thecross-shaped key, and the delete button is pressed, or a taken frame isdropped onto a trash bin icon on the touch panel. In this step S165 itis determined whether or not either of these operations has beenperformed.

If the result of determination in step S165 is that a cancel operationhas been performed, it is next determined whether frame size of a touchoperation is small (S167). In the case where a cancel operation has beencarried out by touch operation, it is determined whether or not the size(horizontal and vertical size, or area) of a frame that should becanceled is small (for example, having an area of 1 cm² or less). Thisis because if the frame size is small it may result in an erroneousoperation.

If the result of determination in step S167 is that the size of a framethat has been subjected to a touch operation is larger than a specifiedvalue, image saving (S169) and update of live view display are carriedout (S171). Even in the case where an image has been deleted as a resultof a cancel operation, there may be cases where it is desired to restorethat image. Therefore, in order to be able to carry out restoration ofthe image (specifically in order to undo a cancel operation), a takenimage that has been designated for deletion (image of a frame of thecombined photograph) is saved in a separate region within temporarymemory, and live view display is updated with that frame appearing in anuntaken state. This image saving will be described later using FIG. 16Band FIG. 16C.

If live view display has been carried out combined photograph creationis carried out (S173). The subroutine for this combined photographcreation will be described using FIG. 11. Since an image that has beendesignated by a cancel operation has been deleted, in this step thecombined photograph is created again using remaining images.

If combined photograph creation has been carried out in step S173, or ifthe result of determination in step S167 was that the frame size with atouch operation was small, or if the result of determination in stepS165 was that a cancel operation was not performed, it is nextdetermined whether or not there has been a restoration operation (S175).A restoration operation is carried out in accordance with operation ofthe operation section 123 or the touch panel 123 a, in cases such aswhen a frame it is desired to restore is selected using the cross-shapedkey and a cancel button is operated, or when a trash bin icon is draggedand dropped to a frame it is desired to restore using a touch operation.

If the result of determination in step S175 is that there was arestoration operation, then next an image is restored (S177), and updateof live view display is carried out (S179). Here, a saved image isreturned to the original position, and live view display is updated withthe appearance that that frame has been taken.

If update of live view display has been carried out, combined photographcreation is carried out (S181). The subroutine for this combinedphotograph creation will be described using FIG. 11. Since an image thatwas designated by the restoration operation has been restored, in thisstep the combined photograph is created again also including therestored image.

If a combined photograph creation has been carried out in step S181, orif the result of determination in step S175 was that a restorationoperation was not performed, it is next determined whether or not therehas been a temporary storage operation (S183 in FIG. 15). In the casewhere there have been a plurality of exposures in a short time andcreation of a combined photograph is performed, shooting can be carriedout continuously. However, in the case of creating a combined photographusing images that were taken at time intervals such as in the morning,in the daytime and at nighttime, it is convenient to be able totemporarily save images during creation of the combined photograph. Withthis embodiment, therefore it is possible to temporarily save imagesthat make up the combined photograph. As a restoration operation,operations such as selecting a temporary save menu and selecting an iconare carried out using a menu screen or touch panel operation etc.

If the result of determination in step S183 is that there has been atemporary save operation, incorporated data is stored (S185). Here, thestate of a currently combined photograph, specifically what photographshave been taken for which style etc., is stored in the flash memory 125or the storage medium 131. As incorporated data that is stored, thereare at least a style (information relating to a template), takenphotograph data, and relational information for the photograph data andthe template.

If storage of incorporated data has been carried out, the combinedphotograph is next reset (S187). After storage, information on thecombined photograph that has been taken is reset to give a state that isthe same as for the first exposure, namely a state where live viewdisplay and the SDRAM 127 have been reset etc.

If the reset of the combined photograph has been carried out in stepS187, or if the result of determination in step S183 was that atemporary save operation was not performed, it is next determinedwhether or not there has been a temporary save read operation (S189). Inthe case where incorporated data for a combined photograph has beentemporarily saved in steps S183 and S185, this operation is the readingout of this temporarily saved incorporated data. As the operation, atemporary save read menu or icon selection operation are carried out ona menu screen or touch panel.

If the result of determination in step S189 is that there has been atemporary save read operation, it is next determined whether or notshooting is in progress (S191). There may be situations where atemporary save read operation is carried out while a combined photographis being taken. In this case, since it is necessary to temporarilyinterrupt the combined photograph shooting that is currently inprogress, this is handled in steps S191-S195. In this step, it isdetermined whether or not combined photograph shooting mode has beenset, and at least one image has been taken for the template.

If the result of determination in step S191 is that shooting of acombined photograph is in progress, whether or not to perform atemporary save is confirmed (S193). In this case, the confirmationscreen for whether or not to temporarily save this current state of theshooting progress is displayed on the display panel 135, as a query tothe user. If the result of this determination is to perform temporarysave, storage of the incorporated data is carried out (S195). Here, thesame incorporated data as in step S185 is stored in order to temporarilysave the current state.

If incorporated data storage has been carried out in step S195, or ifthe result of determination in step S193 was not to perform temporarysave, or if the result of determination in step S191 was that shootingwas not in progress, next the incorporated data is read out (S197), liveview display is performed (S199), and combined photograph creation iscarried out (S201). Here, incorporated data that was temporarily savedin step S185 is read out, update to live view display similar to that instep S171 and step S179 is carried out, and creation of a combinedphotograph similar to that in step S173 and step S181 is carried out.

If creation of a combined photograph has been carried out in step S201,or if the result of determination in step S189 was that a temporary datareadout operation was not carried out, operation of the combinedphotograph operation is completed and the originating flow is returnedto.

In this way, in the flow for combined photograph operation, it ispossible to update a frame to be taken among frames of a style (S163),and it is possible to delete and restore a frame (S169, S177). Further,even during combined photograph shooting, it is possible to temporarilysave data, and it is possible to readout this temporarily saved data,and continue with combined photograph shooting.

Next, the image saving (S169) and image restoration (S177) that weredescribed in the flow for the combined photograph operation will bedescribed using FIG. 16B.

FIG. 16B shows an incorporation image storage region for display andstorage, of the SDRAM 127. The left column of FIG. 16B shows a statewhere three pictures, from within a combined photograph made up of fourpictures, have been taken. Specifically, data for frame image 1-frameimage 3 is stored within the frame image region R1.

In a state where three pictures have been taken in FIG. 16B, if a canceloperation is performed for frame image 2 (Yes at step S165 in FIG. 14),the data for frame image 2 of the frame image region R1 is copied toframe image 2 of the saved image region R2, and data in the frame image2 position of the frame image region R1 is deleted (refer to the middlecolumn in FIG. 16B).

Next, if a restoration operation is performed for frame image 2 (yes atstep S175 in FIG. 14) data for frame image of the saved image region R2is copied to the position for frame image 2 in the frame image regionR1. The data at the position for frame image 2 in the saved image regionR2 is then deleted (refer to the right column of FIG. 16B).

In this way, with this embodiment, a frame image region R1 for storingdata for frame images, and the saved image region R2 for saving of data,are provided. Therefore, if there is a cancel operation data is storedin the saved image region R2, and if there is a restoration operationthe saved data is copied to the frame image region R1. As a result, itis possible to easily carry out restoration even if an image has beentemporarily deleted.

With this embodiment, data is copied at the time of a cancel operationand the restoration operation. This is not limiting, however, and it canalso be handled using pointers. FIG. 16C shows an example of handlingwith pointers.

With the example shown in FIG. 16C, a reference pointer region R3 forstoring reference pointers and a frame image region R1 are provided inthe combined image storage region for display and storage of the SDRAM127. First, at the time point where three out of four images have beentaken, addresses corresponding to frame image 1-frame image 3, as validimages, are stored in the reference pointer region R3 (refer to the leftcolumn in FIG. 16C).

If a cancel operation has been carried out for frame image 2, frameimage 2 remains stored in the frame image region R1, but the address forframe image 2 is deleted from the reference pointer region R3 (refer tothe middle column of FIG. 16C).

Then, if a restoration operation is carried out for frame image 2, theframe image 2 remains stored in the frame image region R1, and theaddress for frame image 2 is restored to the reference pointer region(refer to the right column in FIG. 16C).

Next, detailed operation of the Playback and editing in step S5 (FIG. 6)will be described using the flowchart shown in FIG. 17. As previouslymentioned, this playback and editing is executed when image data thathas already been taken is read out from the storage medium 131, andplayed back.

If the flow for playback and editing is entered, first of all listdisplay is carried out (S211), and image selection is carried out(S213). Here, thumbnails for image files that are stored in the storagemedium 131 are subjected to list display. The user then selects imagesfrom within the list display.

Once image selection has been carried out, it is next determined whetheror not the selected image is a movie (S215). Since still pictures ormovies are stored as images, determination is performed based on theformat of the image.

If the result of determination in step S215 is that the selected imageis a movie, a number of frames is read out (S217). Since a number offrames is stored in the movie file, this information is read out. Acounter i is then initialized to 1 (S219). This counter i counts thenumber of frames.

Next, an i^(th) frame is played back/displayed (S221), 1 is added tocounter i (S223), and it is determined whether or not the counter i issmaller than the number of frames (S225). Here image data for the movieis read out sequentially from the first frame, and displayed on thedisplay panel 135. Movie playback is carried out until the counter ireaches the number of frames that was read out in step S217. After thatprocessing advances to step S227.

If the result of determination in step S215 was that the image that wasselected in step S213 is a still picture, the image is read out (S229),and displayed (S231). Here, image data for a still picture that has beenstored in the storage medium 131 is read out, expansion processing isperformed, and the image is displayed on the display panel 135.

If display has been carried out, it is next is determined whether or nota magnification operation has been carried out (S233), and if the resultof this determination is that a magnification operation has beenperformed magnified display is carried out (S235). During still picturedisplay, for example, magnified display is carried out as a result ofoperation of a magnify button. At this time, in the case of the combinedphotograph, each frame is displayed magnified, or one particular frameis displayed magnified.

If magnified display is carried out in step S235, or if the result ofdetermination in step S233 was that a magnify operation was notperformed, it is next determined whether or not magnification was inprogress and a display position change operation was carried out (S237),and if the result of this determination is that the change operation wascarried out magnification position change is performed (S239). In thecase where magnified display is in progress and a display positionchange operation, for example, operation of the cross-shaped key, iscarried out, a portion within the image that is to be subjected tomagnified display is changed. In the case of a combined photograph, thisis limited to being able to change only the display position within aframe that is being magnified. In a case where there is no restriction,the same display as for a normal image is carried out.

If magnification position change has been carried out in step S239, orif the result of determination in step S237 was that a display positionchange operation was not performed while magnification was in progress,it is next determined whether or not it is a combined photograph and adisplay frame change operation has been performed (S241), and if achange operation has been performed a display frame is changed (S243).In the case of a combined photograph, if magnified display is inprogress and there is a display frame change operation, for example,operation of a function button (Fn button), the frame to be subjected tomagnified display is switched. At this time, a magnification position isstored for each frame, and in the case where there is an instruction todisplay the frame currently being displayed again, the same position issubjected to magnified display.

If display frame change has been carried out in step S243, or if theresult of determination in step S241 was that there was no display framechange operation for combined photographs, it is next determined whetheror not it is a combined photograph and there has been an editingoperation (S245), and if the result of this determination is that therehas been an edit operation, combined photograph editing is carried out(S247). In the case of a combined photograph, by selecting combinedphotograph editing from a menu screen, or selecting an editing icon witha touch operation, it is possible to edit a frame of a combinedphotograph that is currently being played back. Detailed operation ofthis combined photograph editing will be described later using FIG. 18.

Once combined photograph editing has been carried out in step S247, orif the result of determination in step S245 was that it was not acombined photograph and an editing operation was not performed, it isnext determined whether or not display is terminated (S249). Display isterminated if a terminate display button is operated or if a terminatedisplay icon etc. is subjected to a touch operation. If the result ofdetermination in this step S249 is not to terminate display, step S233is returned to and display continues. On the other hand, in the case ofdisplay termination processing advances to step S227.

If the result of decision in step S249 is not display termination, or ifthe result of determination in step S225 was that counter i of the moviehad exceeded the number of frames, it is next determined whether or notto terminate playback (S227). If the playback button is operated oncemore, playback is terminated. If the result of this determination inthis step is not playback termination, step S211 is returned to, andplayback mode continues. On the other hand, if the result ofdetermination is playback termination, this playback and editing flow isterminated and the originating flow is returned to.

In this way, in the playback and editing flow, operations such as movieplayback, magnified display of a still picture, and editing of acombined photograph etc. are executed. With this embodiment, onlycombined photograph editing has been carried out, but besides this it isalso possible to edit movies.

Next, detailed operation of the combined photograph editing of step S247(FIG. 17) will be described using the flowchart shown in FIG. 18. Thiscombined photograph editing, as was described previously, is executedwhen data for a completed combined photographic image is read out fromthe storage medium 131, and an image within the combined photograph isto be replaced with another stored image or a newly taken image.

If the flow for combined photograph editing is entered, first of allreadout of incorporation images is carried out (S251). Incorporationimages are read out, and a main image and subsidiary images are storedin respective storage regions of the SDRAM 127. Here, the main image isimage data of the combined photograph that has been combined, and thesubsidiary images are image data that has been subjected topreprocessing (refer to S123 and S125 in FIG. 11). The subsidiary imagesare stored so as to correspond to each frame in the frame image regionshown in FIG. 16A-FIG. 16C.

Once readout of the incorporation image has been carried out, and editframe is designated (S253). Here, the user designates which, among theplurality of frames that make up the combined photograph, is to bereplaced with another image.

Once an edit frame has been designated, it is next determined whether ornot to replace with a stored image (S255). As was described previously,at the time of editing it is possible to select an image that hasalready been stored or an image that has been newly taken. In the caseof selecting an image that has already been stored, a table of alreadystored images is displayed and an image is selected from among thedisplayed images. On the other hand, in the case of taking a new image,there is a transition to a shooting standby state, and an image isacquired by executing live view display for shooting, and a shootingsequence. Whether to replace with a stored image or to replace with ataken image is selected using a menu screen or icon display.

If the result of determination in step S255 is to replace with a storedimage, list display is carried out (S257), and image selection performed(S259). Here, similarly to step S211 and S213 (FIG. 17), thumbnails forimage files that are stored in the storage medium 131 are subjected tolist display. The user selects an image for replacement from within thelist display by operating the cross-shaped key and the okay button etc.

Once image selection has been carried out, image readout is carried outnext (S261). Here, image data that was selected in step S259 is read outfrom the storage medium 131.

If the result of determination in step S255 is that the image forreplacement is not a stored image, specifically if an image forreplacement is a newly taken image, it is next detected if a releaseoperation has transitioned from off to 1st (S273). Here it is detectedwhether or not the release button has been pressed down half way. Thereis a transfer to the shooting preparation state as a result of pressingthe release button down half way. If the result of determination in stepS273 is that there has been a 1st transition of the release operationfrom off, AE/AF is next carried out (S275). Similarly to step S33 (FIG.7), the AE operation and AF operation are executed here. Once AE and AFhave been carried out, step S273 is returned to.

If the result of determination in step S273 is that there has not been afirst transition from release operation off, it is next determinedwhether or not it is release operation 2nd (S277). Here it is detectedwhether or not the release button has been pressed down fully. If theresult of this determination is that there has not been a releaseoperation second, AE is carried out (S281), shooting is performed usingthe electronic shutter (S283), image processing is carried out (S285),and live view display is carried out (S287). Operation in stepsS281-S287 is the same as was described previously for steps S51-S57 inFIG. 7, and so detailed description will be omitted. Once live viewdisplay has been carried out, processing returns to step S273.

If the result of determination in step S277 is that there is a releaseoperation 2nd, next shooting is carried out using the mechanical shutter101 (S279). Specifically, it is possible for the user to acquire animage that is to replace a frame of the combined photograph by lookingat the live view display and pressing the release button down fully. Ifexposure is finished in the shooting using the mechanical shutter 101,RAW data is acquired.

Once image data has been read out in step S261, or if exposure using themechanical shutter 101 in step S279 is finished, it is next determinedwhether the data format of the image is raw data (S263). The image dataacquired in step S279 is RAW, but the image data that has been read fromthe storage medium 131 includes various formats, such as RAW and JPEG.In the case of RAW data, the basic image processing and special imageprocessing of steps S265-S269 are carried out. In the case of JPEG dataimage processing has already been performed, and so these imageprocessing steps are skipped.

If the result of determination in step S263 is that it was RAW data,basic image processing is carried out next (S263). This basic imageprocessing is the flow shown in FIG. 9, and so detailed description willbe omitted. Once basic image processing has been carried out, it isdetermined whether or not art filter has been set (S267), and if theresult of this determination is that art filter has been set specialimage processing is carried out (S269). This special image processing isthe flow shown in FIG. 10, and so detailed description will be omitted.

If special image processing has been carried out in step S269, or if theresult of determination in step S263 was that it was not raw data, or ifthe result of determination in step S267 was that art filter has notbeen set, combined photograph creation and storage is carried out next(S271). Since image data for a stored image that has been designated asa replacement image is read out in step S261, or image data for a takenimage that has been newly taken as a replacement image is acquired instep S279, in this step a combined photograph is created by replacing animage of one frame of the combined photograph with the replacementimage. Detailed operation of this combined photograph creation and willbe described using FIG. 19.

In this way, with the flow for combined photograph editing, an image forreplacement within the combined photograph is selected from among storedimages (S257-S261), or is acquired by new shooting (S279). Then, a newcombined photograph is created using the combined photograph on thereplacement image (S271).

Next, detailed operation of the combined photograph creation and storagein step S271 will be described using the flowchart shown in FIG. 19. Ifthe flow for this combined photograph creation and storage is entered,combined photograph creation is first carried out (S291). Here, the flowthat was described in FIG. 11 is executed. Specifically, after carryingout preprocessing for each frame making up the combined photograph,adjustment is performed for each frame by carrying out color change andbrightness change etc., so that there is no incongruity. Also, imagequality adjustment is carried out so that differences in image qualitydue to differences in compression ratio etc. are not noticeable. Oncethe color, brightness and image quality of each frame has been adjusted,a combined photograph is created by carrying out combining.

Once combined photograph creation has been carried out, still picturestorage is carried out next (S293). Here, data of the combinedphotograph that was created in step S291 is stored in the storage medium131.

Next, the so-called MPO file in the flow of FIG. 11 will be describedusing FIG. 20 and FIG. 21. FIG. 20 shows the relationship between thefile structure and images stored in the file. This MPO file is stored inthe SDRAM 127 or the storage medium 131. The left side of FIG. 20 showsthe file structure. A main image header is stored in region F1. Asubsidiary image offset representing the address at which the subsidiaryimage is stored, a correspondence relationship between the combiningposition and the subsidiary photograph, and style information for thecombined photograph are stored in this main image header.

Main image data is stored in region F2 of the file structure. As a mainimage there is the combined photograph P1, and this image data is storedin region F2. Also, headers for subsidiary images 1, 2, 3 and 4 arestored in regions F3, F5, F7 and F9, while data for the subsidiaryimages 1, 2, 3 and 4 are stored in regions F4, F6, F8 and F10. Analysisresults of step S125 (FIG. 11), for example, are stored in thesubsidiary image headers.

Main image data that is stored in region F2 is image data of thecombined photograph P1, and subsidiary image data that is stored in theregions F4, F6, F8 and F10 is image data for subsidiary images P11, P12,P13 and P14 constituting the combined photograph P1. In this way, thesubsidiary images (first images), and the combining positions (stored inthe main image header) are stored in the storage section (SDRAM 127,storage medium 131) as a single image file. Subsidiary images (firstimages) and analysis results are also stored as a single image file inthe storage section.

In the case where some subsidiary images constituting the combinedphotograph P1 are replaced, tone and chroma changes carried out usingthe subsidiary images, and the changed subsidiary images are combined.When storing recombined images, images being combined and images beforethe tone and chroma change are saved in the same format.

FIG. 21 shows an example of replacement of the subsidiary image P13 withsubsidiary image P13 a. Specifically, with the example shown in FIG. 20,a combined photograph P1 is created using four images, namely subsidiaryimages P11, P12, P13 and P14. In FIG. 21, a combined photograph P2 iscreated using the four subsidiary images P11, P12, P13 a and P14. Whencreating this combined photograph P2, since tone and chroma are newlychanged using the subsidiary images P11, P12, P13 a and P14, it ispossible to maintain image quality no matter how many times recombiningis performed.

In conventional image editing processing, images that have beendigitized using JPEG etc. are repeatedly subjected to conversion toenhance or soften contrast and chroma with the same number of bits. Inthis case, for example, in the event that numerical values being handledare integers, if multiplication and division are performed values afterthe decimal point are truncated, causing digit dropping of the numericalvalues, with the result that values for gradation etc. are continuouslydegraded, in other words a gradation jump phenomenon arises.

Also, in conventional image editing processing, after carrying outadjustments such as contrast and chroma of a plurality of images,combination is performed to create a temporary combined photograph,followed by combined photograph editing to re-create a combinedphotograph by replacing some images with other replacement images. Inthis case, it is assumed that adjustments such as contrast and chromaare carried out again. If contrast and chroma are repeatedly adjusted,there may be cases where gradation jump arises, and there is degradationin image quality of the combined photograph.

In the above-described conventional technology, if multiplication anddivision are repeated when replacing some of the images constituting thecombined photograph, values of image data after the decimal point aretruncated. FIGS. 22 (t-1)-(t-3) shows an example of returning to theoriginal brightness is returned to after the original image has beensubjected to gradation conversion to darken the image. FIGS. 22(t-1)-(t-3) shows the case where values after the decimal point havebeen truncated, with a portion A being a portion that does not match theoriginal image characteristics, and where a gradation jump phenomenaarises.

In contrast, with this embodiment subsidiary images before creating thecombined photograph are stored independently of the combined photograph.As shown in FIGS. 22 (s-1)-(s-3), after the original image has beensubjected to gradation has been subjected to gradation conversion totemporarily darken the image, the original brightness is returned to andthe two characteristics match.

In this way, with this embodiment, when storing the combined photographa main image is stored as the combined photograph, and images that havebeen resized to the same image size as an image to be incorporated(images whose gradation and trauma have not been changed) are stored assubsidiary images. Then, at the time of editing the combined photograph,subsidiary images are read out, and gradation and chroma are corrected.It is possible to acquire a combined photograph with a natural feel,even if the combined photograph is repeatedly edited. Also, since imagesthat have been subjected to preprocessing such as resizing processingindependent of image combination are stored as subsidiary images, it ispossible primarily to prevent bloating of file size, but is also linkedto a reduction in the replacement processing time.

As has been described above, the image editing device of one embodimentof the present invention carries out first image processing on an imageconstituting a combined photograph to create a first image (S123 of FIG.11), creates a second image, after creating the first image, bysubjecting the first image to second image processing (S129 and S131 ofFIG. 11), creates a combined photograph by combining the second image inaccordance with combining position of the combined photograph (S133 inFIG. 11), and stores the combined photograph and the first image as asingle image file (S137 in FIG. 11). Since not only the combinedphotograph but also the first image are stored together in a singleimage file, it is possible to achieve a reduction in image degradationeven if replacement of some images of a combined photograph arerepeatedly replaced.

The one embodiment of the present invention has been described using anexample where the present invention is applied to a camera, but this isnot limiting, and it is also possible to apply the present invention toan image editing device for creating a combined photograph using alreadytaken images, and to an image editing device that does not have aphotographing function, such as a personal computer.

Further, with the one embodiment of the present invention, a device fortaking pictures has been described using a camera, but as a camera it isalso possible to use a digital single lens reflex camera or a compactcamera, or a camera for movie use such as a video camera, and further tohave a camera that is incorporated into a mobile phone, a mobileinformation terminal (PDA: Personal Digital Assistant), game consoleetc. In any event, it is possible to adopt the present invention as longas a device is capable of shooting combined photographs.

Also, regarding the operation flow in the patent claims, thespecification and the drawings, for the sake of convenience descriptionhas been given using words representing sequence, such as “first” and“next”, but at places where it is not particularly described, this doesnot mean that implementation must be in this order.

The present invention is not limited to these embodiments, andstructural elements may be modified in actual implementation within thescope of the gist of the embodiments. It is also possible form variousinventions by suitably combining the plurality structural elementsdisclosed in the above described embodiments. For example, it ispossible to omit some of the structural elements shown in theembodiments. It is also possible to suitably combine structural elementsfrom different embodiments.

What is claimed is:
 1. An image editing device, for editing a combinedphotographic image formed by combining a plurality of images,comprising: a first image processor for carrying out first imageprocessing on an image and creating a first image; a second imageprocessor for carrying out second image processing on the first image tocreate a second image; a combined photograph processor for creating acombined photograph by combining the second image in accordance with acombining position of the combined photograph; a storage medium forstoring the first image and an image of the combined photograph; and acontroller for, after the first image processing has been carried out,creating a combined photograph by combining the second image that hasbeen subjected to the second image processing in accordance with acombining position of the combined photograph determined by the combinedphotograph processor, and storing the combined photograph and the firstimage as a single image file; wherein the controller, after the combinedphotograph has been created and the combined photograph and the firstimage are stored as a single image file, and a new combined image iscreated by replacing images, which are some of the images constitutingthe combined photograph, with replacement images that are different fromthe first image, subjects the replacement images to image processing bythe first image processor, then subjects the replacement images to imageprocessing by the second image processor, to create a new combinedphotograph in the combined photograph processor using replacement imagesthat are subjected to image processing by the first image processor andthe second image processor, and second images that are obtained bysubjecting the first images other than the replacement images stored inthe image file to image processing by the second image processor.
 2. Theimage editing device of claim 1, wherein the storage medium stores thefirst image and the combining position as a single image file.
 3. Theimage editing device of claim 1, wherein the first image processorresizes an image, or rotates an image, or trims an image.
 4. The imageediting device of claim 1, wherein the second image processor performsat least one of color correction and brightness correction of an image.5. The image editing device of claim 1, further comprising: an analysissection for analyzing features of the plurality of images, andoutputting analysis results, and wherein the storage medium stores thefirst image and the analysis results as a single image file.
 6. Theimage editing device of claim 5, wherein the features are at least oneof a histogram, subject detection results, and size, position and colorcharacteristics of a main subject.
 7. An image editing method, forediting a combined photographic image formed by combining a plurality ofimages, comprising: a step of carrying out first image processing on animage and creating a first image using a first image processor; a stepof, after creating the first image, carrying out second image processingon the first image and creating a second image using a second imageprocessor; a step of creating a combined photograph by combining thesecond image in accordance with a combining position of the combinedphotograph using a combined photograph processor; and a step of storingthe combined photograph and the first image as a single image file on astorage medium; wherein after the combined photograph has been createdand the combined photograph and the first image are stored as a singleimage file, and a new combined image is created by replacing images,which are some of the images constituting the combined photograph, withreplacement images that are different from the first image, subjectingthe replacement images to image processing by the first image processor,then subjecting the replacement images to image processing by the secondimage processor, to create a new combined photograph in the combinedphotograph processor using replacement images that are subjected toimage processing by the first image processor and the second imageprocessor, and second images that are obtained by subjecting the firstimages other than the replacement images stored in the image file toimage processing by the second image processor.
 8. A non-transitorystorage medium for storing a program to be executed on a computer of animage editing device, for editing a combined photographic image formedby combining a plurality of images, the program comprising: a step ofcarrying out first image processing on an image and creating a firstimage using a first image processor; a step of, after creating the firstimage, carrying out second image processing on the first image andcreating a second image using a second image processor; a step ofcreating a combined photograph by combining the second image inaccordance with a combining position of the combined photograph using acombined photograph processor; and a step of storing the combinedphotograph and the first image as a single image file on a storagemedium; wherein after the combined photograph has been created and thecombined photograph and the first image are stored as a single imagefile, and a new combined image is created by replacing images, which aresome of the images constituting the combined photograph, withreplacement images that are different from the first image, subjectingthe replacement images to image processing by the first image processor,then subjecting the replacement images to image processing by the secondimage processor, to create a new combined photograph in the combinedphotograph processor using replacement images that are subjected toimage processing by the first image processor and the second imageprocessor, and second images that are obtained by subjecting the firstimages other than the replacement images stored in the image file toimage processing by the second image processor.